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THE DESCRIPTION AND EVALUATION OF A COLLEGE 
• DEPARTMENT'S FACULTY RATING SYSTEM 



Peter H. Wood 

C/0 Educ. Foundations and Inquiry 
Bowling Green State University 
Bowling Graen, Ohib 43403 



Objectives of the Stu dy • o . k 

This study was undertaken to assess the reliabilities and inter- 
instrument * correlations that characterize various rating procedures used to 
evaluate the faculty of a medium-sized college department. At Bowling Gr^n 
State University, the department chairperson is obligated to assess three 
dimensions of faculty performance: 1) Teaching; 2) Research and/or 
Scholarly Activities; and 3) Service. Results df these evaluations are 
used to determine: 1) Reappointment; 2) Tenure; 3) Promotion; 4) Salary 
Increases; and to some degree, 5) Teacher Assignment. Several evaluation 
procedures^^have been employed during the past three years. "This study 



represents an analysis of some of the characteristics of these procedures. 
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Perspective — Theoretical Framework 

Reduced rates of college expansion and demands for fair personnel 
procedures have caused college administrators to examine their faculty 
evaluation procedures and to make them more objective.^ In the past several* 
years, department chairpersons at BGSU have: bteen sued by faculty claiming 
unfair hiring, retention, or salary policies; been forced to reallocate 
faculty lines due to changes in student enrollment; &»,.d been asked by the 
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Board of Trustees to institute a differential, merit-based system of salary 
increases. Four years ago, the Educational Foundations (EDFI) Department 
at BGSU established a series of committees to investigate faculty evaluatibn 
procedures,. 

The majority of the departmental procedures that were observed seemed 
to be categorizable into one of three types: 

1. Non-Empirical — Administrators and/or selected faculty members 

.•J 

meet in committees to examine vita and make whatever personnel decisions 
are required. • . 

II. Empirical-Ratings — Students, peers, and/or others are asked to 
rate faculty performance on a set of common scales, and these ratings are 
somehow combined with committee or administrative opinions to produce per- 
sonnel decisions. 

ill. Criterion-Referenced — Specific performance criteria are estab- 
lished for IvL^ijij'idual faculty, and faculty are evaluated according to the 
degree to which they meet these criteria. 

This study is a presentation of some of t:he results produced by the 
^atln^-procedur^ -characteristic of the Type II (Empirical-Ratings) approach 
to faculty evaluation. 

Context and Some Limitations 
of the Study 

• Personnel evaluation in a collegial setting — especially in an insti- , 
tution which faces a potential reduction and/or reallocation of staff — 

presents a .wide range of problems. Each evaluation effort threatens some 

I 1 • * 

of those who are asked to support or contribute to it. While increased 

.experience with evaluative procedures and increased pressures to produce an 

objective system creafce a movement toward a more criterion referenced system. 
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general faculty resistance to evaluation for personnel-decision purposes • 
creates a counter force toward a' more casual and less objective approach. 
Much of the data that is missing from th;is report is missing because: 
1) each year the majority of the departmental faculty were supportive of- 
somewhat different procedures 2) individual faculty failed to participate 
in the generally supported procedures because of various logical or ethical 
considerations; and 3) some data — especially information related to the 
rating responses of individual faculty — were intentionally obliterated to 
protect the anonymity of the raters. 

The size of the depattment is small — especially in comparison to the 
number of hypotheses which could be generate d concerning faculty perceptions 
and activities. The rating instruments and procedures fall far short of 
perfection since they were ^nerated more to refect the shifting consensus^ 
of departmental faculty than to reflect the current state of the psycho- 
metric art. Insofar as these measures reilect an Empirical-Ratings stage 

>, • 

of personnel -procedure, one could best describe them ^ "early" or "general" 

' , ■ 'J " ^ - 

ratings. With additional experience with ratings use, it is possiible that 
there may be a shift to more, behaviorally-def ined ratings /Scales aird option 
~keys--prbvlded~"thst~ threre~ls a common agreement as -to^ those behaviors /which— 

represent various degrees of teaching, research, or service performance. 

'. * ' ■ « 

Instruments, Data^Sources and 

Assessment Techniques *- • 

The three primary ^faculty functions of 1) Te^hing, 2) Research/ 
Scholarship, and 3) Service cause faculty to interact with different potential 
raters of these functions.. 
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Student Ratings (1974, 1975, 1976) ; A studdftt rating forrn^ was ^ 
developed and modified over a two-year period. The resulting forpi caused 
students to rate faculty on several dimensions ^(scholarship, organization/ 
clarity, interaction with group, interaction with individuals, and 
enthusiasm)^ to orient them to characteristics of tetrv"alued in a teacher. 
It then asked students to produce a general assessment 'of : 1) the teacher; 
2) the total course ekperience; and 3) their own accomplishment in the course. 
The results of the three general questions were averaged to produce the ^ 
Student rating score. Twenty EDFI faculty were rated by their students at 
the end of the Winter Quarter, 1974. All EDFI faculty , used the same form 
to produce the student ratings scores fqr 1975 and 1976. The score for 
1975 was the total mean score resulting from a'll student responses to the 
three key questiqns for three separate quarters. Spring 1974^ Fall 1974 
and Winter 1975. The°1976 scores were similar ily created. 

Peer Ratings (1974, 1975, 1976) : All- faculty of the EDFI Department 
were asked to rate all other faculty on the three dimensions of teaching, 
research, and. service. The 1974 form consisted of three five-point scales 
in which the one position was 'defined as "low" and the five position was 
defined as "high." The 1975 and 1976 peer forms asked all faculty to rank 



department members from fi?:st up to seventh on each(pf the 'three dimensions. 
The form listed several criteria which were considered to be relevant to each 
dimension. All peers were given access to all department personnel files 
which contained vita, letters of recommendation and other data. Non-ranged 
faculty were automatically assigned the ranking of eight. Tliis ranking 
procedure result^ed from faculty complaints that they could, not honestly rank 
or rate all department members since they were unknowledgeable concerning 



5 



ERIC 



the activities of many. . The 1975-76 peer form produced two stjatistics for 
each faculty member on each dimension—the number of times ranked in the 
top seven, and the total ranking score (with non-rankings equal to eight) . 

Chairperson Ratings (1974) ; In 1974 the chairperson rated all 
faculty on the three dimension, five-point, scale used by pefers. TheM was 
no independent chairperson ranking or rating., in 1975 Or in 1976. 
' . . Committee Ratings (1975) ; In 1974, a faculty evaluation committee 
was created — one membei:,^ elected frcim each of the four departmental ranks 
(instructor, assiistant professor, associate professor, fulj. professor), and 
the fifth person chosen by the department chairperson so as to cause both 
sexes and all departmental sub-divisipfl^ to be represented on the' committee. 
'•In 1975, this committee independently examined the vita of all faculty and 
rated each on five-point scales for teaching, research, .and service. The 

"ive positions on each scale were labeled, and several lead-in questions 

I ■ . 

were used to orient the committee members to criteria believed to be rele- 

vant to each faculty function. There was no committee rating in 1976. 

■ Viisibility ; -Each faculty member was categorized as to yisability to 
other faculty. Faculty wjth offices adjacent to the departmental office 
were labeled as highlj/ visible 1; faculty with offices on the two main cor- 
ridors near the departmental office were labeled 2 for che central corridor 
and 3 for the next most central corridor. ^Faculty in the rear corridor were 
labeled 4; faculty on a different floor of the building were labeled 5; and 
faculty with offices in another building were labeled as 6 (lease visibJLe to 
other departmental members). " ' 

ftank: Faculty were also categorized, as to"'their faculty rank at the 
beginning of each of the three years.. 

Area: Faculty were also characterized as belonging to one of four 



subdivisions existing within the department. 

College Personnel File : Each Spring, ^every department chair is 
requ'ired to file a "Substantiation for Salary and Promotion Recommendations" 
form which present^ the salary, contract type, rank and effectiveness rating 
for each faculty member of the department. All faculty members were rated by 

the chair as to. their Teaching, Hesearch-Service-Schola^ship, and University 

ft 

Service. The labels for the five-goint scale used- for this form are: 
" • 1 = Outstanding 

2 =• Superior 

3 = Above Average . <. , 

4 = Average 

" . 5 Below Average. 
These ratings were available for the 1973-74 year, the 1974-75 year, and 
the 1975-76 year."* Salaries were available for these same years. The three 
ratings for each year were created by the chairperson^ who reorganized the 
various peer and student ratings throughyuse of formulas which shifted 
each year according to faculty or evaluatioa cbnimittee decisions . 

Analysis of Data 

The various ratings and faculty categories were compared through use 
of bivariate cojrrelation analyses — Pearson product-momen-t correlation, Spear- 
man^s rank' orider correlation and Kendall's rank, order correlation (Used 

• ' I mt •k ■ 

If 

ff » > ■ 

when there were many tied ranks). Ratings procedures were analyzed for- relia- 
bility via analysis of variance. Ratings and area, rank and visibility 
identifications were examined via an analysis of variance with area or rank 
or .visibility identification functioning as the independent variables. 



Results . ' ' ^ 

^ . •• . * . 

Colle)^e Personnel File ratings were created by the department chair 

for each faculty member for the* acad<:mic years of 1973-74, 1974-75, 1975-76, 

Although the formulas used to produce these -ratings varied from year to year, 

each was created primarily from some combination of peer and student ratings. 

The student ratj.ngs were blended with peer ratings to produce the Teaching 

scores but. not the Resear'ch/Scholarship or University Service scores,. The 

pattern of Pearson product-moment correlation^coeff icients seems to indicate 

that: 1) the Teacher ratings are relatively consistent across the three 

years — as are most of the Research/Scholairship -and i^ervice ratings; 2) the 

Teacher ratings are generally unrelated to the Resear^ih/Scholarship and 

■* ■ ■ . " 

Service ratings; but 3) the Research/Scholarship and Servicfe ratings are quit 
closely associated with each other. 

TABLE 1 I 

CORRELMIONS BETWEEN YEARLY RATINGS OF THREE FACULTY ^ 
FUNCTIONS: TEACHING (T) , RESEARCH/ SCHOLARSHIP (R) , 
SERVICE (S), (N = 20 to 24)^ 
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68 ^ 
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4. 


1974 


08* . 


26 


.07 
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5. 


1975 


-08 


37* 


12 


69 
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6. 


1976 


-12 


15 


24* 


• 27 . 


62 
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7. 


i'974 


08* 


-05 


" 05> :: 


: 64* 


55 


39 
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8. 


1975 


-06 




20 


62 


n* 


50 






S. 


9. 


1976 


09 


50 


39* 


48 


29 


09*- 


25 





Notes: Correlation cqefficient decimal points have been r^oved, 

Uuderiine4 coefficients reflect a cotmnoh function across years, 

*These correlations reflect a common year but not a common ; function. 
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The Evaluation of the Teaching Function produced the most varied types of 
ratings. In the 1973-74 year, tssaching was rated by 18 of the 24 members of 
the department; by students (for only one terra, Winter 1974); and by the 
department chair. During the following year (1974-75) : most peers indicated 
their rankings (from one to seven) of the best teachers; the five members of 
the evaluation committee rated all faculty on a five poin^ scale; and three 
quarters of sttident ratings were added to the pool. The same (1974-75) peer, 
ratings and student ratings of teaching were again employed the. 1975-76 
academic year. * . 

TABLE 2 

CORRELATIONS BETWEEN' PEER, CHAIR, EVALUATION . 
COMMITTEE, AND STUDENT RATINGS OF TEACHING: 
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Student '75 
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Student '76 




49 
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'64 


,13 


44 
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Peer '75 


39 


-12 


13 


67 










6. 


Peer '76 


35 


18 , 


33 


Al 


76. 








• 7. 


Chair '74 - 


40 ' 


•30 


2:. 


63 


55 


37 






8. 


Committee '75^ 


38 


09 


44 


70 

• 


63 


60 


68 























Notes: Correlation coefficient decimal points have been removed. 
Underlined coefficients -reflect a common type of rater. 
All scales have been converted ti> reflect a similar direction. 
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The consistency of student ratings across tthe three years was not very impres- 

( 

sive. The peer ratings of teaching appear to be somewhat more consistent 
across the three years even though different procedures and forms were 4jsed 
to elicit them during the first year. The committee-peer, committee-chair 
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and peer-chair ratings are similar to the other peer ratings. The relation- 
ships between peer and student ratings for two of the three years is 
similar in nature to the consistency of student ratings across years — hardly 
impressive/ 

Some of the-iow correlations presented in Tables 1 and 2 may be partly 
attributed to a real inconsistency in the beginning performances of faculty 
new to the depart!ment-j-or inconsistency in the way that other faculty per- 
ceive fcheir performance,** When the data associated with the newer faculty — 
those entering the department after 1971 — -are removed from the analyses, 
two new tables are created. 0 ^ 



TABLE 3 



t? 'CORRELATIONS BETWEEN YEARL^Y RATINGS OF THREE FACUI.TY 
FUNCTIONS: TEACHING (T) , RESEARCH/SCHOLARSHIP (R) , 
SERVICE (S). Oig PRE-1972 FACULTY (N=17 to 20)- 



la 

7 8 
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1. 


1974 " 
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2. 


1975 . 


69 ' 
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3. 


.1976 


ii 


68 
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4. 


1974 / 


24* 


20 


15 . ; 
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5./ 


1975 


-05 


• 31* 


05 


65 




R 


6. 


1976 


-23 


-13 - 


-12* 


40 


65 


S 


7. 


1974 


02*^ 


-06 


05 


61* 


55 "■.>3 


S 


8. 


1975 


02 


28* 


07 


52 


75* 44 


S 


9. 


1976 


13 


43 . 


29* 


. 39 


25 -05* 



52 

23 7r 



Notes: • Correlation coefficient decimal points have been removed. 

^ Underline^j:orrelations reflect a common function across yea 

*These correlations reflect a common year but not a common function. 
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TABLE 4 ■ ^ . 

CORRELATIONS BETWEEN PEER, CHAIR, EVALUATION 
^. . COMMITTEE, AND STUDENT RATINGS OF TEACHING: 
• OF PRE-L972 FACULTY (N«16 to 20) o 







» 1 


2 


3 


4 


5 


6 




1. 


Student ''74 
















2. 


Student '75 


65 










A' 




3. 


Student '76 


64 


71 












4. 


Petr '74 


76 


27 


39 V, 










5.'' 


■ Peer '75 


53 


02 


-03 


75 








6. 


Peer '76 


60 


36 


17 


48 ' 


74 






7, 


Chair '74 


60 


35 


18 


11 


11 

r 


, 54 




8. 


Committee '75 


61 


20 


40 


SB 


69 


. 61. • 


67 



Notes: Correlation coefficient decimal points liave been removed 
Underlined coefficients reflect a coramon type of rater. 
All scales have been coxiy^Jtl:ed--^eo reflect a similar direction. 



Comparisons between Tables 1 and 3 seem to indicate that eliminating the 
data from newer faculty has little effect upon the resulting correlations 
among the college ranings. A similar comparison between Tables 2 and 4 
does seem to indicate some change. The three-year consistency of the stu- 
dent ratings improves as does th& apparent relationship between the student 
ratings of the first year the department used a common. form (1974) and the 
various peer/committee/chair ratings on that and subsequent years. ' The' 
Peer Ratings' of the three major faculty functions are presented in 
Table 5. 

Some tentative conclusions* could be developed fron the correlation.^ ^ 
patterns presented in Table sVxi) tjje size of the. correlations between the 
same year's rat^gs/rankings of the three different faculty functions may be 
decreasing as faculty gain experience with identifying and evaluating 
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TABLE 5 



CORREUTIONS BETWEEN PEER RATINGS OF TEACHING (T), 
RESEARCH/SCHpLARSHIP (R), AND SERVICE (S): 
1974 TO 1976 (N = 23 to 26) 
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2 


4 5 6 7 8 -9 10 11 


Peer'lT '74 


It , MM 

\ 






Peer 2 e ''74 


56(54) 




' i 


Peer >S '74 , 


44(43) , 


78(77) 


\ : ■ ■ 0 ■. ■ 










leei:iX'75-L 


^.70(75) 


26(34) 


03(11) \ ' V 


Com 5 T '75 


70(68) 


,52(48) 


30(28)- 66(68) - \ 


Peer 6 R '75 


30(30)° 


^(52) ; 


■-65(65) 42(43) 17(3ip) - 


Coram, 7 R '75 


36(32): 


M(64) 


71(70) 35(36) '54(49) . 78(78) - , 


Peer 8 S '75 


18(18) 


62(62) 


35(35) 34(36) 28(28) 47,(43) 40(36) - ' - . 


Coram 9 S '75 


26(21) 


75(72) 


M(49);;: 28(32). 48(43) 60(57) 65(61) 81(84) - 


Peer 10 T '76. 


,43(48) 


01(04) 


27(27) 78(74) 60(61) 13(12) /l0(05) 06(08) 08(09) - 


Peer 11 R '76 


-07(-23) 20(26) 


35(53) 18(24) 31(08) 39.(39) 49(54) 01(05) 16(11) 32(31) 


Peer 12 S '76 


04(01)" 


39(38) 


.07(05) 36(32) 28(22) 32(25) 30(23) 84(84) 73(74) 23(22) 02(13) 



H 
H 



Notes I Correlatioti decimal points %ve been renoved. , 

Underlined coefficients reflect a common- rating function. 
All scales have been converted to reflect a similar direction. 



.Coefflcleflj:aJ.iL p a r en these ^^^ vilUi Ja la from postrl9?2 faculty eliminated. 
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evidences of these functions; 2) the Research/ Scholarship and the Service 
functions were not clearly ^ff erentiated by peers during the first two 
years of peer evaluatio.ttJLi-S) Temoving the data reflecting the new faculty 

seemed to have little ef f-ect' upon the peer-rating/ranking corrrelation matrix; 

-O^j ! ' ■ 

and 4) correlations are'^light for any one function across more than one 

• y , . 

year. Any 6f thege possible trends would have to persist for several more 
years before they could be described as being more than heuristic hypotheses. 

There are two ways of obtaining peer rankings when you ask peers to 
rank the best seven departmental members as to effectiveness. The peer 
rankings in Tables 2, 4 and 5. were obtained by adding up all of the rank^gs 
for each faculty member.- If a faculty member was not^ranked in the top seven 
faculty members — or was not ranked because another faculty member did not 
participate in the ranking procedure — a ranking of eight was assigned to the 
faculty xmember being evaluated. This ranking of eight was added to the 
other rankings — if any. Consequently colleagues not ranked by anyone' (even 
by themselves) were crediteA^with a peer ranking score of 208 (26 faculty 



members in ±y/:) times b « 208}. The lowest (best) ranking one could achieve 
was a value of 26 — if all 26 faculty ranked you as first. The actual range 



of scores was: 



1975 Peer Teaching: 


151 


to 208, 


mean 




189.3, 


s 


.d. 




18.0 


1975 Peer 


Research: . 


131 


to 208, 


mean 




189.5, 


s 


.d. 




18.5 


1975 Peer 


Service: 


119 


to 208, 


Mean 




189.8, 


s 


.d. 




21.9 


1976 Peer 


Teaching: 


156 


to 240, 


mean 




218.9, 


s 


.d. 




19.1 


-1976 Peer 


Res earch : 


/175 


to 240, 


moan 




219.3, 


s 


.d. 




15.6 


1976 ,Peer 


Service:'? '135 


to 240, 


mean 




"2181 2, 


s 


.d. 




22.0 
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As is obvious from the scores, while 26 faculty were members in the 
department for the Spring ranking of 1975, thirty faculty could have 



participated in the next year's ranking (1976).- Due to a committee decision 
to maintain anonymity of the rankers, the 1976 data was destroyed as soon as 
rank sums were created. Therefore, . there is no additional data for the 1976 
peer rankings. 

* 

The other effectiveness measure created by this • ranking procedure. 
is\ the niamber of times a cplleagua was ranked in the top. seven for one of 
the three faculty functions* The Pearson Product-Moment correlations 
between these two nxmbers — a sum' ranking with nonrankings equal to eight, 
and the number vof times ranked — was -.96, -.98, and -.94 respectively for 
Teaching, Research/Scholarship, ^wid Service. In 1975, twenty-one of the 
twenty-six faculty were fanked in the top seven as teacheris, and twenty- 
^ three of twenty-six were ranked for Research/Scholarship, and for Service. 

Peer Ratings may reflect bias of various sorts. The department is 
subdivided inco four separate areas, and area identification may influence" 
ratings. Different faculty joined the department at 4ifferent times, and 
groups entering during similar periods may form cohorts which influence peer 
ratings. Office locations may influence peer-interaction and so influence 
peer ratings. Unfortunately, peer ratings are anonymous, and the rater 
characteristics are not available for investigaticn. However, the character- 
isZlCB~of the r ated , le e r s ma y^B compart i d with their r at i n gs. — Any discove r ed 
relationships may^ reflect bias — or they may reflect a reasonable and logical 
relationship with performance levels. Table 6 presents some of the relation- 
ships between ratings or rankings and area, year jpined department, a^d 
office locSLtion. 
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TABLE 6 . ' 

KLATIONSHIPS BETWEEN STUDENT (S) , PEER (P) AND COMMITTEE (C) 
• . RATINGS .AND AREA, YEAR ENTERING DEPARTMENT, 
AND OFFICE LOCATION' • 



Rating? Source 



AREA 



F (df) Eta^ 



YEAR ' 



F,. (df) Eta^ r 



OFFICE 



F (df) Eta^ r 



P Teaching '74 

P Research. '74 

;P Senice '74 

S Teaching ' '74 

P Teaching '75 

C Teaching '75 

P , Research '75 

C Research .'75 

P Service '75 

C Service. • '75 

P Teaching. '76* 

P, Research '76 

P Service '76 

S Teaching '76 



1.25 (3,18) =.17 

.52' (3,18) .08 

2.0 (3,18) ;25 

.33 (3,14) .07 

1.71 (3,21) .20 

.53 (3,21) .07 

.70 (3,21) .09 

;.9S':.(3,21) .12 

.39. (3,21) .05 

.32 (3,21) .04 

\.55 (3,20) .19 

.60 (3,20) .08 

.51 (3,20) . .07 

2.29 , (3,21) .25 



1.75 (13,9) .71 , -.03 2.40 

.62 (13,9) .48 -.20 1.42 

.72 (13,9) .51 -.08 1.14 

1.1^(10,8) .60' -.22 .68 

2.96' (14,11) .79 -.05 10.46 

1.37 (14,11) .64 .02 3.39 

.88 (14,11) .53 .11 • 2.56 

1.29 (14,11). 62 .08 1.78 

.67 (14,11) .46 .12 , 2.36 

.63 (14,11) .44 .13 2.84 

3.89 (13,11) .82 .00 3.41 

1.24 (13,11) .59 > .06 l;i9 

■.35. (13,11) .29 .07 .76 

.66 (14;i0) .48 -.20 1.00 

A ■ 



5,17) .41 .44** 

5,17) .29 .10 . 

5,17) .25 .19 

4,14) ',16 .09 . 

5,20) .72 .58*** 

5,20) .46 i37 

5,20); .39 .37** H 

5,20) .31 .29 . 

5,20) .37 .34** ' 

5,20) .42 .14, 

5,19) .47 '!37** 

5,19) .24 . .21 

5,19) .17 , ^23 

5,19) .21 -.05 



Not^i: "F" is the ANOVA "F" ratio of lean squares. ' " , 
"r" is the Kendall correlation coefficlent.- 



' **Correlationt coefficient is significant beyond the. .01 level, 
***Correlation coefficient is significant beyond the .001 level, 
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The data presented in Table 6 can be interprated as an ♦indication that 
office location — or visibility to bther faculty — might bias peer ratings of 
teaching in favor of those faculty which have offices in ateas which are more 
centrally located within the distribution of departmental offices. Visibility 
may also influence peer ratings of research arid service. There are faint hints 
t'at: 1) the ^eer ranking approach used in 1975 and 1976 may be more open to 
visibility bif j; and 2) experience with ranking of p^ars may reduce this 
'Visibility" bias. Student ratings, of teaching seem relatively unrelated to « . 
office location and to area identification but are slightly related tc the year 
that faculty began teaching in the department — with the more experienced 
teachers eliciting slightly higher ratings. Some of this relationship may be 
related to the increased power to teach graduate students or preferred classes 
that xaky be gained with longevity within the department. ^ ^ 

Data concerning instrument reliabilities is now being collected. Table 7 

. ■ ' ^ ■ i ■■ ■ 

•presents some of the data collected from some of the procedures. VThe^ "relia- 

bility" figure was derived from a r n mpay-t gnn^ nf ^hp mpan CTitn nf nqnnrnn hotwnn 

teachers and the mean sum of squares within each teachef 's ratings or rankings. 

The^ormula used is; the reli&b.ility_estimate-(x) =_CF-l)7-F-i— ^ ^~ — ~ 

The popularity of the departmental evaluation system seeT::£d to be rela-' 
tively low. Tlie peer rating system used in the sprli^g of 1974 (aIX"'fa iuity ^ 
rate all. other faculty) was voted out in the fall of 1974. The peer ranking 
system used in 1975 and in 1976 has yet to be voted out of use, l^ut a depart- 
mental vote in 1975 caused the separate rating by the elected, five person. 

Faculty Evaluation Committee to be eliminated. The most recent departmental 

» • • 

vote was quite. strongly in favor of increasing the participation of the 
departmental chair in the evaluation of faculty. The same departmental vote 



■TABLE 7 

RELIABILITY ESTIMATES OF VARIOUS DEPARTMENTAL 
RATING AND- RANKING PROCEDURES 







Number 

Ol 

Raters 


Anova 
F Ratio 


Estimate of 
KeiiaDiiity 
(r=(F-l)/F)= 


Student /Ratings: Fall 1974 


1609 


13.00 


.92 


Peer 


Racings, '74 Teaching 


20 


2.61 


.62 


Peer 


Ratings, *74 Service 


20 


2.71 


.53 


Peer 


Ratings, '74 Research 




. 2.68 


.63 


Fac. 


Evar. Connnittee Ratings, '75 Teaching 


5 


3.83 


.74 


Fac. 


Eval. Committee Ratings, '75 Service 


5 


6.31 


.84 . 


Fac. 


Eval. Committee Ratings, '75 Research 


5 


5.08 


.80 


.Peer 


Rankings, '75 Teaching '75 , 


18 


4.64 


•78 


Peer 


Rankings, '75 Service '75 




5.61 


■ .S2 


Peer 


Rankings, "'75. Research '75 


18 


5.54 


^ .82 



was also in favor of reducing" the wQighl:^x>f ~^the peer rankings ai^d of incf easing 
the weight given to student ratiuj^s of teaching. For: ^he^past~-four years, the 



university has requested colleges .and departments to provide some sort of 
evidence that faculty merit was being identified and rewarded a t_t_he_dep.art.-- 



mental level. Much of the previously described effort was in partial response 
to this request. A reduction in univerisity pressure ml^ht easily r*^-'^^^^ "^"^ 



elimination of all peer or student ratings or rankings — at least for^ tenured' 

. " - • • • «^ . « ' 

faculty who.^re not within one year, of promotion. 

■ * ■ ' ' 
Conclusions . . " , ^ / 

VJhen the number of analyses exceed the number of subjects, any cpnclu- 
sions must be regarded with considerable caution. The following conclusions 
therefore ara categorized as: (I) Tentative; arid (II) Very Tentative • 
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Tentative conclusions: • . 

1) Student ratings of teaching do not parallel faculty ratings or rankings 
of teaching — possibly because different criteria are applied by ea^ 
group; 

2) Student ratings of teaching are relatively stable across a three year 

peflod — for experienced faculty; 

■J . ■ ' • ■ 

3) Peer ratings or rankings of t;eaching are also relatively stable-,-if 

^• 

less so that student ratings-'-but may be influenced by non-Heaching- 

related variables such as faculty "visibility"; 

4) Peer ranking systems which permit peers to rank only the "better" faculty 
are preferred by faculty to any system which requires faculty to tate 

or rank all faculty of a 20 to 30 person department; 

5) Such peer ranking may produce ranking with a consistency (reliability?) 
at least as good .as that characterizing an "all rate all" system; 

6) A peer committee may produce ratings which are similar in nature to 
^ the rankings produced by an entire department. 

Some of the more tentative conclusions are: . 



1) Faculty with little experience in rating or ranking their colleagues may 
find it difficult to differentiate between' the different faculty " 



functions which are broadly labeled as,: Teaching; Research/Scholarship; 

and Service; ' 
^Z) Rater or ranker ability 'to differentiate between these different 

. futvctions may improve with increased experience; , 
3) The ini^^ial publication of student ratings of teaching — or any other 

indication ofs^f ectiveness — may influence faculty evaluations of 

teaching (or^othei\functions) for several successive years; and 



A) The instutlon of a formal, faculty evaluation system tikL-II stimulate 
many faculty to develop a wide variety jjf methoas by which they can 
inform other faculty about an. incredible variety of previously un- 
heralded activities. This last comment is not supported by the data 
already pres'ent, but is believed to be true by -;ost members of the 
department. ' 
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Notes/References 



guibn, Hutchinson, Klein, Statz and Wood have just completed 
a year-long ^survey olBGSU faculty attitudes toward the 
evaluation of faculty. The preliminary results seem to 
indicate that the majority of faculty are generally in 
opposition to external evaluation of their efforts. 
surprising result was that students were preferred to peers 
and chairpersons as evaluators of teaching pi^rformance. 
This report- is as yet unpublished, but will be submitted 
to ERIC in the near future. 

The student rating form used in this study is presented in 
the appendix of this report. The first five questions were 
' adapted from the genei^al factor titles developed by Hildebrand, 
Wilson and Dienst and reported j,n their Evaluating University 
Teaching (Center for Research and Development in Higher 
Education, University of California, Berkeley, 52 pages, 
undated). The questions actually used to prod^ice an evalua- 
tioh cf the teacher's classroom effectiveness ar^ the three 
Very general, judgemental questions which follow the first 
five Berk^eley-derived, orienting questions. 

In general, those correlation coefficients larger than .39 
tend to be significantly different from zero at the .05 leyel , 
those higher than lAQ B,t che .01 level, and those higher than 
.7 at the .001* level. , Although levels of, significance vary 
slightly dUie to changes in number of cases, these figu7:es : 
provide a useful and general rule of thumb for all of the 
correlations used in* this report.. ^ 



Research by Sullivan aiid Skanes (validity of student 
evaluations of teaching and the characteristics of success- , " 
ful instructors. Journal of Educational Psychology , 1974 » 
66 pages, 584-590) has provided evidence of the lack of 

consistency o f the student ratings' of relative ly in^Yp^T-i^. 

eucud teachers, unpublished work at BGSU .with the ratings 
and student test scores of graduate-assistant teachers has 
also indicated a considerable lack of consistency of graduate- 
assistant teachers froifl term to term. 

Winer, J., Statistical Principles in Experimental Design , 
Second Edition, McGraw-Hill Book Company,.. 1971, ^pages 283- 
287. - 
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APPENDIX 

Page A2 of this appendix is a copy of. the letter sent to all depart- 

<^ ' 

mental faculty to introduce the 1975 peer ranking system. Page A3 oresents 

■ ; . • . ■ • " . 

the criteria t*or each faculty function to be. ranked. Pages A4 to A7 present 
the student rating instrument and its "direction sheet;^. 

The" rating procedure used in the Spring of 1974 and the Faculty Evalu- 
ation Committee (1975) rating procedures were similar in that all faculty and 
the five FEC members were provided with a list of all. faculty and a name for 
the three faculty functions ^Teaching, .lasearch/Scholarsbip, ^Service) and were, 
asked to rate each faculty member on a 1 to 5. scale with "5" representing 
superior or excellent function and "1" representing poor performance. 



.=^rf^==-^ A2 

I Tr^^rni I „ ^ . . Department of Bdiicatronal 

\uLJf==\ Jl Bowlipg Green State University Foundations & Inquiry 

.^^Si:^=^= Bowling Green, Ohio 43403 

^ • (419) 372-0':51 ext 32Z 

V ' " ■ ■ 

April 29, 1975 ■ " ., 
MEMORANDUM 

TO: . EDFI Faculty ' ' 

. ■■ ■ ■ ' . <i ■ . ■; ■ 

FROM: EDFI Faculty Evaluation Committee 
RE: ' Pepr Tnput to Faculty Evaluation 

After considerable, discussion, the EDFI F;3^culty Evaluation* Committee has 
decided, that peer input is an important — and unique — source of informatiotf 
relative to'decisions fconcerning EDFI faculty. The peer input procedure 
used last ye&r — everyone rate everyone — hsis too maay obvious logical and 
psychometric disadvantages. ThI system in which «*ach faculty member asks' 
cieveral peers to provide ratings^-recommendations also has many disadvantages. 
' A third approach combines simplicity with psychometric and logical reason- 
ableness — ^while still producing a type of peer opinion likely to be a 
valuable supplynent to student ratings, chairnp.rson ratings and- committee ; 
opinion* 

Our department is *50 liirge and its members* interests and accomplishments 
are 30 diverse - thai: it is unreasonable to believe that all of us are 
awar« of the coatributlons and istrengths of all members. However, pur 
facv^i'jy are making valuable* contributions ift the a^eas of TEACHING and/or^ 
REuEARCH and/or SERVICE, and these contributions —'manv: j>'f which are not 
adequately represented in vita or known to all of us — ^'are -known ^o. some of ' 
their* colleagues . This knowledge can be transmuted into input to FEC 
decisions via the following coUeague-percsption-of-contribution system. 
Each of the following three pages provides a set of questions and/or 
statements which* partially define one of the three areas of academic 
contribution—Taaching, Research/Production, and Service. Each page also 
contains a list of EDFI faculty.* Faculty members are asked trt: . 

^ \ _ (1) diBcide upon their own definition of teaching (or research or. 

^ ' —service)-^- - - 

(2y indicate which faculty member — t:o-iheir_toowledge — best exemplified 
this def initiok during the past year : 

(3) indicate this person by writing the number "1" in the space^^ext^ 
to that person's name; 

(4) indicate v7ho is the next best exemplar of their definition by 
placing a number "2"; and 

(5) continue this procedure until a minimum of two faculty and'a--'^ 

. mayinwim number of seven faculty are ranked on each of ^e three 
areas of contribution (faculty, of course, may nominate themselves 
^in the position that they consider most appropriate). 
* . • ' 

Faculty resumes^ for most (many) faculty are available in the departmental 
office for those who'wish to view theln. Please return these formis to 
Cathy Long next week — May 5 to May 9. ^ 
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TEACHING 

1) Effectiveness in stimulating students, to learn 

7J) ' J&iowledge of content area - * 

3) , 'Effectiveness in sharing tea^ching competencies with colleagues* 

4) 'Efforts to improve teaching effectiveness . f * • 

5) Effective'ness in adv^^ment 

6) Supervision of thesis, diissertation, and/or independehr study 

7) Development of innovative courses or programs. v 



SCHOLARLY OR CREATIVE EFFORTS 

(Publications, Programs, Research) * ' 

1) Has the necessary competencies tJ produce scholarly or creative Efforts 

2) Develops proposals, publicationfl^, papers, programs, presentations \ 
-3)_ Slgr?,?;ficant in 'influence on faculty, organizations, school systems, 

programs ... . * . 

4) Improves the quality and quantity of scholarly creative efforts by 

interaction with other faculty,,- attendance at workshops or professional, 

meetings, extensive reading • . . ' * 
5^ Functions as a consultant ; • 



SERVICE 

1) Is an active and valuable contribxitor to university committees or 
groups (at the area, department, college, and/or university or state, 
level) " ' 

2) Provides service to .peers and colleagues * ^ 

3) Is an active and valuable contributor to professional associations 

4) Provides professional public service beyond this campus assist, other 
universities ,v colleges, schools, agencies, companies — (not including' 
"good citizenship" activities performed in. the capacity of a concerned 
citizen in church, youth groups, etc . ) * • • : 

5) Has received special awards and/or recognition for professional service. 
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' ^ • ■ SnmEST D£SCSI?TI0tT '07 mcssTG ■ ' 

• 2^ 7°^ for.wias cie STUDSET DESCSETION 0? TZAC23G. Tor each class, ycu vill ' 

(1) . class quaacicias of the q^iescionnairs; 

(2) class quaacities of ■ cha ZZH ')555 ansver sheac, s=d 

(3) oca copy or zhls xora ed be ca=oleted by the ceacher of each section. 
^Ihe procadura for 'fora use involves: 

• (1) giving the foms, and ansver shaecs (and soca pencils) co a sccdit ac Che ■ 

beginniag or a- class period (BUT NOT DURING TE2 FIsnTSdyJ": * • Cr® 

board[^^ "^^yf^ "^^^ coursa nane and nuaber, and s'eccion nt=ber on tSe black- 
Che Jlis LTSy':hri:^f'.^ ..cu^ent reads- the directions to the class .. and ; j : : 

• (4) adding- your data sheet (on the>back of this page) to ■thi- stack of class 
forns and answer sheets, and asking a student to «ail-(cLpus nall).^r ^et-Ir5"CoT 
hand>. an foms to: Cathy Long, Dept. SDrl, BGSU (529 SdiSationTiuSpJ)^^^ .■ 

Ve have begta the nse of an optically -scanned ansver sheet in order t/ avoid thre- 
v^.ek" keypunching ^elay that ve faced in the past. Since ve vill sc^^ 
a>)l(s to use the process con? uter pro grans, developed last tern, we hofce to be"able to' 
return the results early enough to be of use to you for your ia::r t/S-rclLses! 

^ cote about tie iom: 

. Sage 1 Quescdocs 1 to ,5 rapres^at the general teacher qtlallities.-osc ^-eGue-tr- 
S-:S.^o'^"^^""r"-5 °' teachers'- by- BGSU -studentl-STw conese'^-ide::^ 

.-a over 40 years of student ratings research. Questions 6-9 are ve^r geae-al cuesc^L 
reflecting the 'student-perceived eff ectivecess of ^the class, ilncel^^ ^JSaiv of suS - 

Soae of the other queitions o'a' pags-l reptrasenFth^^ 
ceri3cij^ni?h-^7--cause ratings to be biased unward or dcviiva-i ^t-ti ^4lst^ll ■ 
:-Cl^8 the general problen related fo' the fairness* of conp4ri-g^I-ie iSd c?Sf H ^ 

undergraduate and graduate classed, etc.. A fev of t^.e^ oSLlCfst^as ^Slte 
course clarity or difficulty and shonld be of interest to^ >5st S^Sty? . 

.^th questions on the back page are intended to suonly the ceache- \ 

^^th^nore specific inromation about the class, if.- you vish Co eliiLLta^anr(oJW- 
iLt T 'l"^'=C°'^> ^° ^7 including chis request in the directions to be r- ad to 
your cl«s^or by not printing the back page vhen you reproduce the ^m? If ^ou " 
Vish to acd your- ovn- questions, do so 'a? ha-rtLng your student =iss then cut -? -he 
forn — anc nodify the directions to indicate this - or by pr^nti-'z c'-^n ol -^ ' 
It^^T, " piece of ours. If you use our ^Z^^^^^ ' ' 

?lea«e start nunbering.your ova questions— if dlffarent --on ou-s- ••-H — 

44 and. finish.vich nunber 70 so that - i:s vers to your o-I-; q;7es=ions -^iL -o^^i::^ ■ ^ 
conrusec vita these of other c.aachars vho use cur questions. 

Ve viU process, and return all dasa as/soca as possihle. - ' ' ' ' 



